Skip to content

feat: one-shot verified, compressed database backup into any directory - #26

Merged
zebraengine merged 1 commit into
mainfrom
feat/backup
Aug 27, 2026
Merged

feat: one-shot verified, compressed database backup into any directory#26
zebraengine merged 1 commit into
mainfrom
feat/backup

Conversation

@zebraengine

Copy link
Copy Markdown
Owner

Problem

The database is the history — the charger keeps none, and the degradation watch judges every new charge against it — yet nothing backs it up. The docs say "back up that one file and you have everything" and leave the how to the reader; on the reference install that meant one 1.4 GB file on one NVMe, no copy anywhere. Roadmap #1 "Automated backup helper", scoped down on purpose: the user asked that nothing run on a schedule unless they say so, and that storage location be entirely theirs to choose.

Code touched

  • wallmonitor/backup.py (new) — run_backup(db_path, dest_dir, compression, keep): SQLite online backup API into a temp file beside the source (never in the destination, which may be a sync-watched folder), PRAGMA integrity_check on the snapshot with a hard failure if it isn't ok, compress (gzip default / xz / none, stdlib only), fsync, atomic rename to wallmonitor-<serial>-<UTC stamp>.db.gz, then rotate. rotate() is pure and judged by the timestamp in the filename (sync clients rewrite mtimes): newest per day / ISO week / month for D/W/M, a file kept by any tier is kept, 0/0/0 rotates nothing, and only names matching its own pattern for this serial are ever returned — a second charger's files or anything foreign in the folder are untouched. Serial comes from the device_serial setting, sanitized, unpinned on a fresh database. The snapshot inherits WAL mode, so its -wal/-shm siblings are swept with it.
  • wallmonitor/config.py--backup DIR (WM_BACKUP_DIR), --backup-compress (WM_BACKUP_COMPRESS), --backup-keep D/W/M (WM_BACKUP_KEEP, validated at parse time). --backup is exempt from the --host requirement, like --compact and --discover.
  • wallmonitor/__main__.py — one-shot branch mirroring --compact: run, print sizes and timings and what was rotated out, exit 0.
  • deploy/install-service.shopt-in --backup-dir DIR writes <service>-backup.service (oneshot, same user, same WM_DB) and a daily .timer (03:30, 15 min jitter, Persistent=true). Without the flag nothing is scheduled; --uninstall removes both units.
  • docs/running.md — flag rows and a Backup section: what it does, measured numbers, that the directory is the user's decision (four examples, none preferred: second disk, NAS mount, sync-client folder, a directory another machine pulls from), that nothing is scheduled by default, and a restore recipe that removes stale -wal/-shm — an old WAL applied to a restored file corrupts it. docs/recording.md and README point at it.
  • tests/test_backup.py (new, 7 tests) — snapshot round-trip (integrity, row count, serial, live DB still writable, no temp files left in either directory), unpinned serial and none/xz compression, rotation on synthetic names (40 dailies + same-day earlier copy + other-serial + malformed + foreign files) and on disk (foreign file survives, exactly the newest three remain under 3/0/0), a corrupt source is refused before the destination is touched, and the flag needs no --host while a bad --backup-keep is a parse error.

Risk

  • The temp snapshot is a full uncompressed copy beside the source — 1.4 GB of transient disk on the reference install, removed in a finally. A box with less free space than its database cannot back up; the error is a plain OSError, and the destination is never touched.
  • The online backup API copies in one pass from a read-only connection; the live poller keeps writing (measured: no poll gap, no offline). The copy reflects a moment early in the pass — for a telemetry database that is exactly what a snapshot means.
  • Rotation deletes files. It is scoped by serial and by exact filename pattern, and 0/0/0 disables it, but a user who points two chargers with the same pinned serial at one folder (impossible unless one is a clone) would have them rotate against each other.
  • The installer timer runs as the service user with the service's WM_DB; it does not inherit the service's other environment (it doesn't need any). If --db was not given at install, the default wallmonitor.db in the monitor directory is used, matching the service.
  • No pull client, no cloud code, no schedule by default — deliberately. rsync from another machine is a documented example, not a feature.

Verification

  • python -m pytest -q: 120 passed (113 + 7 new).
  • Live run on the reference install, branch code, against the production database while the service polled: 1395.9 MB snapshot (integrity ok, 3.5 s) -> 110.7 MB gzip (14.8 s), 21 s wall. No .wallmonitor-snapshot* left beside the source, no .tmp in the destination, service offline: false throughout.
  • Pulled to a second machine with rsync -a (6 s on the LAN), gunzip (1.4 s), PRAGMA integrity_checkok, journal_mode wal. Row counts against the live /api/status counts taken seconds later: vitals_samples 1,549,943 vs 1,549,953 (the ten rows polled after the snapshot), wifi_samples 134,375 = 134,375, sessions 60 = 60, events 2,904 = 2,904, alerts 82 = 82; pinned serial present; newest sample timestamp equals the snapshot stamp to the second.
  • Earlier measurement on the same database: xz -6 reaches 64 MB in ~4 min; documented as the option for the patient.

Deploy: git pull on an install; no service restart needed (one-shot CLI, service behaviour unchanged). The installer's --backup-dir only matters on a re-run of the installer.

`wallmonitor --backup DIR` snapshots the live database with SQLite's
online backup API (consistent under WAL, no service stop), runs
integrity_check on the copy and refuses to continue unless it is ok,
compresses it (gzip default, xz, or none), places it atomically as
wallmonitor-<serial>-<UTC stamp>.db.gz, and rotates by the date in the
filename (newest per day/week/month for 7/4/12; 0/0/0 keeps all). Only
its own files are ever deleted. The snapshot temp lives beside the source,
never in DIR, so a sync-watched folder never sees an uncompressed copy or
a half-written file.

Where DIR is — local disk, NAS mount, a sync client's folder, a directory
another machine pulls from — is the user's decision; wallmonitor never
talks to anything beyond it, and nothing is scheduled by default. The
service installer gains an opt-in --backup-dir that adds a daily timer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@zebraengine
zebraengine merged commit dc03319 into main Aug 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant